<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=GBK">
<title>Lab #2: Lexical Analyzer</title>
<link rel="stylesheet" type="text/css">
</head>

<body>
<font face="Trebuchet MS">
<table width="100%">
<tbody>
  <tr>
<td>
<h2>Lab #2: Lexical Analyzer</h2>
<hr>

<h3>Introduction</h3>

<p>This lab requires you to write a lexical analyzer for the 
MiniJava language as described in the Tiger book. In order to do 
this, you will need to figure 
out all of the tokens that you need to lex, by reading pages 
484-485 of 
<a href="http://staff.ustc.edu.cn/~bjhua/courses/compiler/2011/tiger.chm">
  <i>Modern Compiler Implementation in Java</i></a>. </p>

<p>
In this lab, you will be building your lexer using ML-Lex. 
There is online documentation on 
<a href="http://www.smlnj.org/doc/ML-Lex/manual.html">ML-LEX here</a>. 
There is also some help in chapter 2 of 
<i>Modern Compiler Implementation in Java</i>
</p>

<h3>Start</h3>

<p>
To start, you should download this 
<a href="http://staff.ustc.edu.cn/~bjhua/courses/compiler/2011/labs/lab2/lab2.rar">
package</a>,  
in which we have offered you a code skeleton. Here is a brief 
discription of the files:
</p>
<table valign="top" border="1" >
  <tr>
    <td >sources.cm </td>
    <td >makefile</td>
  </tr>
  <tr>
    <td>Test.java </td>
    <td>a MinJava program used to test your lexer (you may
    also want to add your own test cases)</td>
  </tr>
  <tr>
    <td>errormsg.sml </td>
    <td>definition of structure ErrorMsg, members of this
	structure will be used in your lexer</td>
  </tr>
  <tr>
    <td>java.lex </td>
    <td>The source file of lexer, it will be used by ML-Lex 
	to generate the lexer</td>
  </tr>
  <tr>
    <td>runlex.sml </td>
    <td>the entry point of whole program</td>
  </tr>
  <tr>
    <td>tokens.sml </td>
    <td>definition of tokens data structure</td>
  </tr>
  <tr>
    <td>tokens.sig </td>
    <td>signature of structure Tokens</td>
  </tr>
</table>
<h3>Your Job</h3>
<p>
Your job is to edit the code in <code>java.lex</code>. You will 
have to remove some of the sample code and add a lot of your 
own. The tokens are declared in <code>tokens.sig</code>. Here 
is a part of list of symbols that will appear in the source along with 
the tokens they should be associated with:
</p>
<table valign="top" border="1">
  <tr>
    <td>symbol   </td>
    <td>token   </td>
  </tr>
  <tr>
    <td>=</td>
    <td>ASSIGN</td>
  </tr>
  <tr>
    <td>&amp;&amp;</td>
    <td>AND</td>
  </tr>
  <tr>
    <td>!</td>
    <td>NOT</td>
  </tr>
  <tr>
    <td>&lt;</td>
    <td>LT</td>
  </tr>
  <tr>
    <td>(</td>
    <td>LPAREN</td>
  </tr>
  <tr>
    <td>*</td>
    <td>TIMES</td>
  </tr>
  <tr>
    <td>+</td>
    <td>PLUS</td>
  </tr>
  <tr>
    <td>,</td>
    <td>COMMA</td>
  </tr>
  <tr>
    <td>)</td>
    <td>RPAREN</td>
  </tr>
</table>
<p>
Keywords should be represented using tokens with the same 
name. The end-of-file token should be represented using the 
token EOF.</p>

<p>
Each token takes two integers: the line number and column 
number of the beginning of the token. These are used for error 
reporting. In the example below, <code>A</code> is on the second line in 
column 9. Notice, the first row is row 1 and the first column 
is column 1 (as opposed to 0).</p>

<pre>
      x = new A ();
</pre>

<p>Use <code>ErrorMsg.error : ErrorMsg.pos2 -> string -> unit</code> 
to report errors. It takes two arguments: a pair of 
file-positions (the beginning and end of the erroneous text, 
measured in characters from the beginning of the file), and 
the error message to print out. You should keep the 
<code>ErrorMsg</code> module informed of where the newlines 
occur (by calling <code>newLine</code>) so that it can translate these 
file-positions to line numbers.</p>

<p>The <code>make_pos</code> function is a convenient way to 
convert the things ML-Lex knows (yypos and yytext) into the 
file positions of the beginning and end of the token.</p>

<p>Be careful with your syntax in lex files. Remember that each 
lex definition must end with ";" and each lexing rule must also 
end with ";". If you forget ";" then ML-Lex will complain. For 
example, the following is correct:</p>
<pre>      
... 
fun eof () = ... 
...

%% 
alpha = [A-Za-z]; 
... 

%% 
"class" => (Tokens.CLASS (make_pos(yypos,yytext)));
...
</pre>

<h3>Compile, run, and test</h3>
To run your code, you should first run 
<pre>
ml-lex java.lex
</pre> 
to generate your lexer, whose name will be <code>java.lex.sml</code>, and then type 
<pre>
CM.make "sources.cm";
</pre>
to compile the program. Finally, run 
<pre>
RunLex.runlex "Test.java"; 
</pre> 
It will output a sequence of tokens along with the line and 
column number for each token. <code>"Result.txt"</code> gives the lex result of <code> "Test.java"</code>.

<p>
As always, a single test is far 
from complete. You will want to write your own test cases and 
thoroughly test your lexer. Repeat the above procedure once your modify
your code.
</p>

<p><B>Nested comments.</B> This assignment would be a lot easier 
if the MiniJava language didn't have nested comments. I 
recommend that you first get everything working except for nested 
comments. Then add that feature as your time permits.</p>

<h3>Handin</h3>
This completes the lab. Hand in your solution to information system.

</td>
</tr>
  
</tbody>
</table>
</font>


</body>
</html>
